Searching for Robust Chaos in Discrete Time Neural Networks using Weight Space Exploration

A method for analysis and synthesis of recurrent discrete time neural networks with robust chaotic behavior is presented. Using this method, a new nonlinear activation function called saturated modulus was found to be the most efficient in order to get chaos even in small sizes neural networks. Based on weight space exploring using a searching strategy closely related with genetic algorithms, some new concepts were introduced, namely the generic structure of a neural networks population and the concept of descriptor map. Instead of trying to learn a specified (chaotic) trajectory, entropic and sensitivity maps are computed and displayed for a population of neural networks sharing the same generic structure. The sensitivity map is the particular case of a descriptor map based on Liapunov exponents. These maps can be used to select from a population of neural networks those individuals which best fits with some desired behavior. Moreover, using sensitivity maps, robust chaos was emphasized meaning that for some compact set included in the weight space the chaotic behavior of the network remains unchanged. Experimental results also proved that the activation function used by neurons is strong related to the global aspect of the descriptor maps and they can be efficiently used for synthesis.